Vision system for viewing a sporting event

ABSTRACT

The systems combine images of a real scene with computer generated imagery where the computer generated imagery is particular to the position and pointing attitude of the device. Being useful to spectators of the game baseball, the device can produce images showing where a strike zone is relative to the position of the players; can produce images of ball tracks; of replays and many other forms of information having characteristics dependent on a viewer&#39;s position and attitude.

This application is a continuation-in-part of a previously filedapplication having a Ser. No. of 08/119,360 filed on Sep. 10, 1993, nowU.S. Pat. No. 5,815,411 issued Sep. 29, 1998.

BACKGROUND OF THE INVENTION

The instant invention is generally concerned with an electro-opticvision system which exploits knowledge of position and attitude todisplay information in a perspective unique to that position andattitude, and is specifically concerned with a vision system for use ina sporting event to view various features associated with the game froma users perspective. This disclosure relies on the teachings of theabove-identified application therefore the entire content of thatapplication is incorporated herein by reference. As preferredembodiments are directed to the specific game of baseball and manyreferences are made thereto, the "Rules of Baseball" as set forth by theCommissioner of Baseball are assumed to be well known by those familiarwith the art. It should be understood that all of the herein referencedmaterials are provide considerable definition of elements and featuresof the present invention.

Currently, electronic devices to aid in the comprehension andunderstanding of the game of baseball are being tested in stadiumenvironments. Particularly, a device called "SportsTrax" made byMotorola is being tested at the Toronto Blue Jays games. Users canprogram an audible alerts that indicate the end of an inning, a runscored, a home run, and when a rally is taking place. If an opposingplayer belts one out, a crying sound is heard. Because data is enteredfrom the press box and transmitted to users after a two minute delay,the usefulness is limited for spectators present at the game. Thetechnology is crude and has been adapted from the pager technologiesknown in messaging arts. Whereas baseball is a fascinating sport withgreat opportunities of pleasure for spectators in attendance at thegame, systems of the present invention are configured to enhance visualaspects of the game.

SUMMARY OF THE INVENTION

Comes now, an invention of a vision system including devices and methodswherein a computer generated image is merged with an image of a realscene such that a composite image is presented in proper perspective andaligned to the real scene as the scene is viewed from the location ofthe user. Devices of the invention being primarily characterized byhaving a camera for collecting an image, a computer, a data store whichis responsive to a position determining means and an attitudedetermining means, and a display for presenting an augmented image to auser.

A camera can be used to form an electronic image of a particular sceneof interest, the image having the identical perspective as the scenewould appear to have to a person viewing the scene without a vision aid.Using image processing routines, a computer can be used to manipulatethe electronic image before it is presented at a display. The originalimage may be altered according to many known technologies such as colorenhancement, edge improvements, anti-vibration schemes, etc., but moreimportantly, the original image may be augmented with information from adata store where the information is particular to the scene beingaddressed as defined by the position and attitude of the vision system.Information is arranged by the computer to appear in the properperspective and is combined with the image of the real scene. Theinformation which relates to the scene can be embodied in many forms,but is generally related to certain spatial features of the scene. It iscritical to note that the information relates to a particular scene, andis merged with a real electronic image of the scene by a computer suchthat the perspective of the displayed information corresponds to theuser's perspective and is aligned to the scene being addressed.

Some types of information that may be merged with an image of a realscene being addressed include historic, statistical, symbolic, abstractsymbolic, action tracks, histogram of action tracks, computer graphicaluser interface devices. Each of these types, can be combined with animage of a real scene in a spatially dependent fashion to contribute tofurther understanding of the scene being addressed. We call thecomposite image that has been augmented with information from memory:"Augmented Reality". To present a basic understanding of the variousinformation types, a simple example of some of them follows; a completeand detailed explanation appears in the detailed description portion ofthis disclosure. Images of real scenes can be combined with recordingsof images and image series from past times which share same or similarspatial features of present scenes. A real scene can be improved byinserting statistical information relating to spatial features of thescene. Symbols to represent otherwise invisible image features of ascene can be added in appropriate locations. Similarly, symbols torepresent abstract geometrical features such as property lines or planarboundaries can also be added to real images. Motion can be illustratedby highlighting a path across a scene to show where a moving object hastraveled. When an object takes a similar path in a series of relatedmotions, a history of those paths can be presented together to form ahistogram type plot. As with many computer systems, user inputs can bevia a computer graphical user interface means such as "windows", "pointand click", "click and drag". A special form of window we call a "Panel"is used in some versions of the invention. Panels are used for userinterface and are sometimes related to spatial features of a scene.

Although many sporting events have a large spectator participation, thepresent disclosure is directed to the particular features of baseball.It will be easily appreciated that the devices and methods taught herecan apply to other sports without deviation from the spirit of theinvention. A better understanding can be had with reference to thedetailed description, preferred embodiments and the appended drawings.These embodiments represent particular ways to realize the invention andare not inclusive of all ways possible to benefit from the invention.Therefore, there may exist embodiments that do not deviate from thescope and spirit of the invention as set forth by the claims, but do notappear here as specific examples.

Wherefore the demands are high, the invention answers those demands withthe following objects:

It is a primary object of the invention to provide a vision system foraugmenting real images.

It is also an object of the invention to provide a vision systemoperable for combining computer generated images with real images wherethe computer generated images are presented in a perspective whichcorresponds to the user's perspective of a scene being addressed andfurther being aligned to the scene being addressed.

It is a further object to provide a vision system which combines theelectronic image from a camera with information recalled from a datastore where the information from the data store is particular to theposition and attitude of the camera.

It is another primary object of the invention to provide a vision systemfor use in sporting events such as baseball.

It is a further object of the invention to provide a vision system forviewing a live baseball game where images from the game are augmentedwith a computer.

To arrive at these and other objects, features and advantages, deviceshaving:

an electro-optic vision system for viewing a sporting event comprising:a camera for producing an electronic image of a scene being addressed,position and attitude determining means for determining the position andattitude of the camera, a data store responsive to the outputs of saidposition and attitude determining means and having information storedtherein, computer operable for combining the stored information with theelectronic image to produce an augmented image, and an electro-opticdisplay operable for displaying the augmented image;

and methods having the steps of:

producing an electronic image of a real scene with a imaging device,determining the position and attitude of the imaging device, recallinginformation from a data store which depends on the position and attitudeof the imaging device, augmenting the electronic image with theinformation from the data store, are provided.

BRIEF DESCRIPTION OF THE DRAWINGS

The features, aspects, and advantages of the present invention willbecome better understood with regard to the following description,appended claims and drawings where:

FIG. 1 is scenario showing baseball fans viewing a game through systemsof the invention;

FIG. 2 is a close-up showing the augmented images of FIG. 1;

FIG. 3 is an image of a baseball field and two composite images asviewed by two users;

FIG. 4 is a block diagram of a version of the invention;

FIG. 5 shows an image of a real scene and a computer generated imagecombined to form a composite image;

FIGS. 6, 7, and 8 show a simple geometry appearing in three differentperspectives;

FIGS. 9, 10, and 11 show a detailed scene as it relates to baseball inthree different perspectives;

FIG. 12 shows an image of a player as recorded by a camera and acomputer generated graphic superimposed thereon;

FIG. 13 shows a "Panel" placed in the field of play by the computer;

FIGS. 14A and 14B show abstract graphic data superimposed onto realfield images;

FIG. 15 shows ball tracks;

FIG. 16 shows a flow chart hierarchy for a version of the invention;

FIGS. 17-19 are composite images of the invention;

FIGS. 20-97 are flow diagrams of a preferred embodiment. Each figureafter 20 is a subset thereof. The titles of each figure indicate a levelwithin the flow diagram hierarchy.

DETAILED DESCRIPTION AND PREFERRED EMBODIMENTS OF THE INVENTION

The baseball system is a dynamic, innovative system that allows a fan toobserve things they otherwise couldn't see, understand things they mightnot otherwise comprehend, and greatly enrich the game experience. Thesystem exploits data related to the user's position and orientation togenerate graphics that are combined with real images derived directlyfrom a real world view. Using this system, the user would be able to seeand, in some cases, interact with graphical representations that allowthe user to see things that are not there or are obstructed. The systemmay have various elements in communication with each other where eachelement may be located remotely from one or more of the other elements.An important element of the system is a hand held instrument which theuser interacts with is sometimes herein called a "Brighteyes" unit or"hand held unit". Similar in size and weight to an electronic camcorder,it is generally in communication with remote computers and also has somecomputing facilities therein. FIG. 1 shows two fans 11 in a similarlocation within the stadium but looking in different directions. Thescene that a first fan might see is an outfielder throwing a ball 12.The scene the second fan may be viewing is the play at second base 13.It is fundamental to the invention that the system recognize what sceneis being addressed. To accomplish this, the hand held device 14 has areference viewing axis associated with it. The pointing direction ofthis viewing axis of the first fan's unit 21 is pointing in a differentdirection than the viewing axis of the second fan's unit. The unit has afacilities to determine the pointing direction of the unit. The positionof the fans within the stadium is also measurable. Having position andattitude measurements, a computer can determine what scene is beingaddressed by each fan. The computer can then respond by augmenting anelectronic image made with a camera with information generated by acomputer in accordance with the position and attitude measurements.

System Hardware Configurations

For baseball, or similar applications, a "Brighteyes" unit 14 issupported by data services that are accessed by wireless means. Suitableradios, with repeaters if required, are commercially available. Theradio provides communication between the Brighteyes unit and largercomputer systems having larger computing and memory capacity. Therecould be a large number of users present at the baseball park and thatthese users could be making simultaneous use of Brighteyes systems. Tosupport this large number of users, the radio communication system mayoperate by broadcasting some information. We divide the computer intosub-systems based on function. A broadcasting sub-system is hereincalled an "Action Data System 46". Certain computer requests andservices would also be available on a single user request basis. Singleuser requests are handled by a "Transaction Data System 48".

Therefore, in preferred versions there are three computer sub-systems: afirst which is a battery operated Brighteyes unit held by a user; asecond which is called an Action Data System 46, a computer sub-systemrequired to track and broadcast the position of the ball and all theplayers, managers, umpires and coaches involved in the play to theBrighteyes units; and a third which is called a Transaction Data System48, a computer sub-system that provides statistics, information andother services on a broadcast or request basis to the Brighteyes units.Each of these computer sub-systems may be in communication with one oreach of the others.

Brighteyes Hand Held Units

Best versions of the hand held battery operated Brighteyes unit contain:an input lens assembly 41 with auto focus and zoom and a color CCD 44whereby real images are acquired; an attitude sensing system 43 thatprovides information on the pointing direction of the Brighteyes unit;user input keys whereby the user of the Brighteyes unit may interactwith or input information to the unit; a two way radio whereby theBrighteyes unit may interact with and receive information from, and giveinformation to, remote computers and other Brighteyes units; a pair ofcolor liquid crystal displays with viewing lens assemblies such that theoutput images may be observed by the user of the Brighteyes unit; acomputer that has access to: the video stream from the electronic cameraassembly; data from the digital radio; keyed input from the user;attitude information from the attitude sensing system and position data.The unit may be equipped with an audio in/out system enabling speechprocessing to input spoken information to the Brighteyes computer. Thismay be in addition to or in substitution for other means for informationinput within the preferred embodiment. The computer in the Brighteyesunit also has access to the lens assembly for the purpose of sensing thezoom and focus position of the lens and for the further purpose ofcommanding the lens to zoom and focus as required by the control programrunning on the Brighteyes computer and in response to requests from theuser. The computer within the Brighteyes unit 45 can input images fromthe real world from the lens assembly and augment, delete, andsubstitute information in graphical form before presenting the augmentedcomposite images to the user at the display.

The computer within the Brighteyes unit 45 contains a small amount ofnon-volatile read only storage in which a unique identification numberis stored during manufacture. Also within this read only storage is acontrol program that when accessed by the computer at power up causesthe Brighteyes unit to initialize and commence operation. The computerwithin the Brighteyes unit 45 also has random access memory and,read-write non-volatile storage (for example battery backed up CMOS RAMor a hard-disk drive) for the storage of software and for the storage ofinformation between games. For purposes of cost, size and reduction inpower consumption, it would be possible to dispense with this read-writenon volatile storage by increasing the amount of software within theread-only storage to include the capability to initiate data transferover the radio and by depending on the remote services from theTransaction Data System 48 described below for the down-loading ofadditional software to the Brighteyes units and for the storage ofinformation on behalf of specific Brighteyes units between games. It ispossible that a number of different versions of the Brighteyes unitscould be operating at any particular event. Since each unit has a uniqueidentification number and can make this number available over thedigital radio to the Transaction Data System 48 it is entirely possiblefor the Transaction Data System 48 to provide customized services to alarge variety of different Brighteyes units of varying configuration.Since the exact location of the seat within the stadium occupied by theuser can be uniquely identified from the user's ticket required forentrance to the park and the X, Y, and Z position coordinates of theseat in relation to a reference datum (the "Stadium Datum") can betransmitted to the Brighteyes unit by the Transaction Data System 48 thepreferred embodiment of a Brighteyes unit especially configured for thisapplication does not require a position sensing system. An alternativemeans for positioning the individual Brighteyes unit using triangulationto two points whose position is accurately known to the Brighteyes unitis discussed later and may used in alternate versions. In the event thata Brighteyes unit containing a position sensing system such as GPS wasbrought to and was operable at the game then, providing that Brighteyesunit was equipped with a two-way digital radio and was registered withthe stadium, the user of that GPS equipped unit could wander at willthroughout the stadium and make use of all the services described inthis application from any viewpoint. Similarly, Brighteyes unitsequipped with additional optional equipment such as the capability torecord and play back video information, and or with an optional higherspeed radio making use of a TV sub-carrier (e.g. TVT1) could operate atthe stadium or ball park providing only that those Brighteyes units wereequipped with a suitable digital radio and were registered with the ballpark or stadium such that remote data services could be requested andprovided.

Within the random access memory of the Brighteyes unit, would be theoperating software of the Brighteyes computer unit and a variety of datavariables defined below. (Many other variables would also be stored suchas: the control policy for the radio, the lighting threshold to enableblanking of the flood lights, user selected information).

Within the random access memory of the Brighteyes unit would be adigital representation of the stadium ("Stadium Model") stored in such away that the X, Y, Z positions and identities of every significantnon-moving object and surface can be determined by software within theBrighteyes unit accessing the digital representation. Additionally thedigital representation would provide information enabling physicalsurfaces such as the walls and playing surface to be differentiated fromopen spaces such as the sky. This Stadium Model could be downloaded tothe Brighteyes units as periodically broadcast from the Transaction DataSystem 48 or it could be stored between games within the non-volatileread write storage if fitted.

Within the RAM of the Brighteyes unit would be the position in terms ofX, Y, Z of the ball in play and each player, manager, coach and umpiretaking part in the current play. We refer to these as "Action DataElements". In the game of baseball there are generally 21 Action DataElements at any one time (see below). The Action Data Elements positionand movements would be broadcast to the Brighteyes units by the ActionData System 46. Each Action Data Element would have a unique identifierto enable retrieval of further information relating to that Action DataElement. Upon receiving the X, Y, Z information based on the stadiumzero datum the hand held unit would convert the data to its own zerodatum. The unit would do this by subtracting its own X, Y, Z (Xu, Yu,Zu) from the X, Y, Z (Xe, Ye, Ze) of each Action Data Element astransmitted by the Action Data System 46. Thus the calculationsXe-Xu=Xn, Ye-Yu=Yn and Ze-Zu=Zn give a new set of coordinates (Xn, Yn,Zn) with the position of the hand held unit as the new zero datum foreach Action Data Element. Other methods of translation of axes may beequally effective. The unit would also use the same calculations toproduce its own location specific version of the Stadium Model andgenerate new coordinates for any graphics requested from the TransactionData System 48.

The location of the Brighteyes unit in terms of X, Y, Z (Xu, Yu, Zu) inrelation to the Stadium Datum such that the position of the Brighteyesunit may be accessed by the software within the Brighteyes computer. Thepointing attitude of the Brighteyes unit stored such that the headingand attitude of the Brighteyes unit may be accessed by software withinthe Brighteyes computer. The focus and zoom position of the lens issimilarly stored so that both the focus of the lens and the zoomposition of the lens may be accessed by the software within theBrighteyes unit. Using the heading, attitude and location of the unitand the focal length and zoom position of the lens the field of viewwould be calculated for the unit at that instant and the scene beingaddressed is then better known to the computer.

The X, Y, and Z position of an Action Data Element in relation to theunit zero datum of the designation point on the boresight of theBrighteyes unit could be calculated by using the position andorientation data of the unit to project the boresight axis of the unitforward while searching that part of digital model of the park, ActionData Elements and objects within the field of view as calculated aboveto find the point where the boresight axis intersects with the object,surface or Action Data Element. The user may put the cross hairs of theunit on a particular Action Data Element, surface or other image objectand by pressing a "designate" key may select the item so selected forfurther processing. Multiple items may be selected and variablesrelating to such designation such as location and time of designationrecorded within the Brighteyes unit for further processing. To calculatethe coordinates of the designated point in terms of the stadium zerodatum the unit would add its own Xu, Yu, Zu to the coordinates of thepoint thus reversing the process.

Action Data System 46

The purpose of the Action Data System 46 is to track and identify thelocation of each Action Data Element and to broadcast that informationin real-time by wireless means to the Brighteyes units.

In baseball examples of the Action Data Elements are: the ball in play,each player on the field (nine defense and a maximum of four offense),two base coaches, four umpires, and one manager or pitching coach.Typically there are 21 Action Data Elements in baseball.

The preferred embodiment for the Action Data System 46 is an industrystandard personal computer with: at least 66 Mhz clock rate; two "framegrabbers"; a two way digital radio; a keyboard; a high resolution colorscreen equipped with a light-pen or other pointing system to enable anoperator to rapidly select items on the screen. Connected to the twoframe grabbers would be two high speed color video cameras. Optionaldisks and peripherals could also be attached providing the primarymission of the Action Data System 46 was not reduced by bandwidthcontention.

The two high speed video cameras could be located in fixed andcalibrated positions within a stadium to enable the position of everyAction Data Element to be in view. Each camera would be connected bylinks to a frame grabber within the Action Data System 46. Each framegrabber would enable the computer processor in the Action Data System 46to use stop frame action to track the location of each Action DataElement in view. Additionally, the Action Data System 46 could receive afeed from the ball speed radar typically located immediately behind homeplate.

As each Action Data Element entered the field of play it would beidentified to the software of the Action Data System 46 either bychecking off a pre-arranged list or roster or by the operatordesignating a player, umpire, manager, coach or umpire using thelight-pen and indicating by keystrokes a unique identifier. The ball inplay would also be designated in this way. Using the unique identifierthe Action Data System 46 would have access to data relating to eachplayer to allow the image processing software in the Action Data System46 to better analyze the stop-frame images of the game and, whererequired, to enable the operator to resolve difficulties in machinerecognition of play. Such information would include the color of theuniforms, the height of the player, left or right handed in play andother information. Typically the operator would be required to designatethe general area where the ball would first appear from the pitcher'shand so that the image processing software could commence tracking theball in play rather than those in use in the bullpens or other visuallysimilar and confusing objects such as sea birds or garbage on the field.The operator may also be required to re-designate players where two ormore players crossed and/or doubled back at a plate or elsewhere on thefield.

Alternative technologies exist for various versions of the trackingfunction of the Action Data System 46 including the use of: precisionradar, laser scanners, radio determination using passive or activetransponders, and other more exotic techniques using interferometry. Itmay well be that a combination of techniques could be used to determinethe position and identification of each Action Data Element and that amore automatic and possibly more effective method may be introduced.

Having determined the position and identification for each Action DataElement the Action Data System 46 transmits this information over thedigital radio to the Brighteyes units and to the Transaction Data System48.

The Action Data System 46 would also transmit results of the previousplay as it relates to statistics. Each piece of information relating toa specific player would be tagged with a personal ID number. Theseresults would be received by both the Transaction Data System 48, whichwould update its database to include the new information, and theBrighteyes hand held units themselves if the unit had, at that time, anyof the different statistical lists of the players in question displayed.Software in the Brighteyes hand held unit would use this new data toupdate the displayed statistics, thus saving the bandwidth necessary totransmit the players entire statistics after each play.

Transaction Data System

The purpose of the Transaction Data System 48 is to handle all of thenon-real time processing required to support the remote Brighteyesunits. Unlike the Action Data System 46 the Transaction Data System 48is equipped to respond to individual requests from the Brighteyes units.It is also equipped to broadcast information, typically between innings,to all Brighteyes Units. Furthermore, the Transaction Data System 48 isequipped with long-haul data links to access systems in other stadiumsand in other facilities.

A version of the Transaction Data System 48 is an industry standardpersonal computer in a configuration similar to the ADS but with thefollowing exceptions and additions. The Transaction Data System 48 doesnot have frame grabbers and associated video cameras. To store the largeamount of information required, the Transaction Data System 48 isequipped with large memory. In addition to provision for one or moreradios the Transaction Data System 48 is equipped with data links givingaccess to: other stadiums; ticketing systems; credit card centers; wireline news systems; gaming services; transportation information anddispatch systems; and potentially other remote data services.

The Transaction Data System 48 has access to an extended version of theStadium Model. The Stadium Model is a specialized data-base and includesinformation relating to the location in terms of X, Y, and Z of everysignificant non-moving object and surface within the stadium. Thestandard Stadium Model is the version that can be transmitted and storedin each Brighteyes unit. The extended version of the Stadium modelincludes the position of every seat in the Stadium and the location ofevery: concession; fire hydrant; electrical junction box; and otherinformation relating to the operation and management of the Stadium.From this extended version of the Stadium Model it is possible for theTransaction Data System 48 to provide customized services either byradio to standard and specialized version of Brighteyes units (forexample for fire-fighters, security services, and stadium maintenance)or by means of printed or other reports. Additionally the extendedversion of the Stadium Model could be made available such that ticketingoutlets could be equipped with color or monochrome CRT or other screensto display the exact location and a digitized version of the view fromthe seat being sold or offered at point of sale. The extended andstandard version of the Stadium Model could be created and updated fromexisting plans of the appropriate Stadium. By using a Brighteyes unitand the designate capability outlined above it would be possible for themanagers of the Transaction Data System 48 to readily update the StadiumModels with additional or changed information.

The Transaction Data System 48 would have a data-base containinginformation relating to all aspects of the game of baseball. Majorleague baseball currently maintains a very comprehensive data-base offacts relating to all aspects of the game and this information, updatedby the Action Data System 46 after each play, would be available via theTransaction Data System 48 to Brighteyes units either on a broadcast orindividual query basis.

A Brighteyes unit within range of a stadium radio system after power upwould send a request to the Transaction Data System 48 for the currentradio schedule and for the frequency policy. In addition to informationrelating to channel allocations, frequencies and protocols to be usedthe radio schedule would define the times and on which channel theStadium Model; software and software updates; initializing statistics;and other information would be broadcast. In parallel with thistransaction the Transaction Data System 48 would refer to the data baseof Brighteyes units and would establish ownership identification eitherfrom the data-base or by query to the Brighteyes unit that the userprovide additional identification. If the Transaction Data System 48could not verify that the Brighteyes unit was in the hands of aregistered and authenticated user then it would alert stadium securitybut otherwise proceed as normal.

Following user and Brighteyes unit identification and authentication,the Transaction Data System 48 would query the ticketing system forinformation relating to that user and would establish, as in the case ofa season ticket holder, if a seat number had been issued and was knownto the system. The Transaction Data System 48 would verify with theBrighteyes user that the issued ticket was the seat number to beactually used or would request from the user for the seat number to beused. To facilitate correct input of the seat number (e.g.: URES28 row:15; Seat: 20) the ticket could be encoded with a bar-code or othergraphic and by allowing the unit to "see" the ticket the informationcould be acquired from the ticket when held in the appropriate positionwith relation to the input lens of the Brighteyes unit. Alternativelythe ticket number could be entered by the user using keys on theBrighteyes unit. Once the seat number is known to the Transaction DataSystem 48 a look-up of the information in an extended version of theStadium Model will provide the position of the seat in relation to theStadium Datum. This information would then be transmitted to theindividual Brighteyes unit for storage within the Brighteyes unit and inthe case of a mislaid or stolen Brighteyes unit the information would betransmitted to Stadium security. There are other ways in which anindividual Brighteyes unit could establish accurate positioning with theStadium. It would be possible to use the digital radio system to locatea Brighteyes transmitter by means of radio determination. Given themulti-path problem associated with a large enclosed area with many radioreflective surfaces this determination of location would be non-trivialand would potentially either lack in sufficient accuracy or may take anundesirable length of time. However, for a stationary Brighteyes unit itwould be quite possible to locate the unit with sufficient accuracy thatfine tuning of the position could be accomplished by the user enteringjust part of the ticket number (e.g. instead of URS28, Row: 15; Seat 20simply use Row: 15; Seat 20) or by means of triangulation where theBrighteyes user would "designate" two points known to the TransactionData System 48 which could then calculate the position of the BrighteyesUnit and transmit that back to the unit. Alternatively the individualBrighteyes units could calculate position from similar triangulation totwo points in the Stadium Model. Exploiting triangulation in this latterfashion may indeed be chosen as the preferred embodiment. Additionally,a GPS equipped Brighteyes unit could operate within the Stadium, even ina covered stadium, assuming pseudo-satellites could be positioned withinthe Stadium or differential GPS was available in a suitable form.

The Transaction Data System 48 maintains a data-base of Action Data fromprevious plays and can broadcast replays either recorded from the ADS atthe stadium or received by communication link from the Transaction DataSystem 48 at other stadiums. In the case of replays from other Stadiumsor recreating scenes and events from the past the Transaction DataSystem 48 would preface the replay by broadcasting the stadium Model forthat remote Stadium or, on the assumption that there had been physicalchanges to the Stadium in the interim, for that historical play. Thesereplays could be either in the form of stick-figures or could includecomputer generated animation of the players in action. The TransactionData System 48 maintains a data-base of animated information. Computeranimation of replays and other events could be provided either bybroadcast using the digital spread spectrum radio and/or at someincreased bandwidth to suitably equipped Brighteyes units.

Information Displayed in Perspective

Having the above described hardware in place and operable, the systemcan now address a scene, collect an electronic image of that scene 52,identify the scene, generate information relating to the scene 51,combine the computer generated information with the image to form acomposite image 53. It is particularly important to consider thecomputer generated information. It is known to display "low battery"indicia in prior art systems but the system of the invention considersinformation which is particular to the scene being addressed beingidentifiable to the computer by determination of the position andattitude of the device.

It is further important to note that "low battery" indicia do not haveassociated therewith any particular perspective. In contrast, theinformation of the invention typically is spatially dependent andtherefore has associated therewith a particular perspective. FIGS. 6, 7and 8 show a geometric shape in three different perspectives. It thencan be appreciated from FIGS. 9, 10, and 11 that each seat of a baseballstadium represents a different perspective of the field. Compositeimages generated by systems of the invention are presented in aperspective associated with a particular user and are aligned to thereal scene such that the normal direction of the composite image planeis in the pointing direction of the camera.

Special information types and features include: historic, statistical,symbolic, abstract symbolic, action tracks, histogram of action tracks,computer graphical interface devices, et cetera.

History

With systems of the invention it is possible to combine images fromother time periods onto live images of real scenes. Historic recordingscan be played-back with live scenes. Also, computer generated"futuristic" scenes can be combined onto real images.

If a user of the vision system desires to see how it appeared when HankAaron broke the career home runs record in Fulton County Stadium in1974, the images from that time can be recalled from the data store,adjusted for proper perspective according to the users position andattitude, and replayed onto the field as the field appears today. Areplay of a film simply shows the perspective as it was recorded on theday of the actual event. Because that perspective may have no relationwith the viewers present perspective, the replayed film is like simplevideo play-back. To give a strong feeling of "being there" systems ofthe invention align the events of a historic event to the field as itsurrounds the user. If the user chooses to watch the ball fly over thefence, then the camera is pointed into left field as the ball fly'sover. If the user decides instead to watch the reaction of Hank Aaron,the device is pointed to the home plate--first base line. In eithercase, the viewer "sees" the historic event superimposed with the presentstadium.

In similar fashion, events that have not yet happened but that aregenerated artificially can also be "replayed" onto present scenes. Thiscan be used to "imagine" "what-if" scenarios.

Representation of Statistical Data

Statistical data can be represented in many forms. "80% of all 0.300hitters are heavier than 220 pounds" is a simple numeric (80)representation of information regarding a player's weight. Sometimesinformation can be presented in a graphical form which relates the datato the subject of the data. For example an outline of a player's figurethat is 80% shaded on the inside. Presentation of data in relation toimages may give the user a better feeling of its significance and may beeasier to interpret.

When statistical data relates to a geographic area, the data may bepresented superimposed onto the geographic area as one views it fromvarious perspectives. By using the system of the invention, one looksinto a display having an image of a real scene aligned to the positionand attitude of the user with respect to that scene. The perspective ofthe scene as viewed normally looks the same as the perspective of theimage displayed when the system is pointed towards the scene. An area ofinterest within the scene may have statistical data associated with it.In this case, the display can superimpose indicia onto or around thatarea in a predetermined way to represent that data.

Considering FIG. 6, we can imagine looking down on a right triangledefined by the perpendicular black lines. The shaded area could indicatesome high probability density or other statistical feature. If one movesfrom viewing the triangle from directly "above" or normal to the planeof the paper, then the perspective of the triangle changes FIG. 7. FIG.8 shows how the same triangle looks at a glancing angle off to the rightside of the apex. The shaded area, perhaps representing a regionassociated with some statistical information changes as well. In allthree perspectives, the shaded area indicates a particular area ofinterest. It is the same area in all three figures.

This can be further illustrated by considering what a baseball fieldlooks like from each seat in a stadium. In baseball it is common topresent statistical data associated with a portion of the field. If aparticular batter frequently hits to left field, a display can bearranged to indicate the trend.

Users of the system are distributed throughout the stadium seats andeach has a different perspective of the field. By looking into thedisplay of their Brighteyes system with the device pointed in thedirection of interest, the Action Date System could instruct the unitsto shade the left field as defined by a particular set of coordinates.When the coordinates are plotted, the resultant area will appeardifferently for every Brighteyes unit.

As a new batter that is up to bat is identified, the computer could findthat the batter has hit the current pitcher to left field 90% of thetime. The Action Data System 46 then transmits an instruction to shadethe left field by defining the region with three points. The handheldunit processors receive the shading instruction that corresponds to thestatistical information and the coordinates of the area that will needto be shaded. Each unit being located in a unique position with respectto the left field, will uniquely represent the left field on its displayin accordance with the perspective as viewed from the user's location.Although the left field is a simple two dimensional area, statisticalinformation can also be represented in many three dimensional forms.

This can be further illustrated by considering a players defensive rangein a baseball scenario. If we choose a probability threshold of a playerresponding to a ball in play as a function of position near the player,and indicate where that threshold lies, the limit would surround theplayer in the shape of some dome like form (FIG. 12). Extending perhapsfurther in the direction ahead of the player than in the directionbehind (this is due to the fact that a player can run much faster in theforward direction than in the reverse direction). Whereas a player wearsa glove on one hand, there may be some bias to either side. (backhandplays are more difficult than open handed plays.) This could bepredicted by a history of plays made for a certain player or by averagesfor a certain league or team. The dome, being determined by whatevermeans, would be embodied by a some surface with the baseball playerinside and the baseball field completing an enclosure. This enclosurewould appear differently from every seat in a stadium. If the drawing inFIG. 12 existed as a printed image in a game program with an explanationof a centerfielder's range, it would be extremely difficult for a fansitting in the outfield bleachers to translate where on the field in thedirection of second base corresponds to the limit of the fielders range.However, if the data of FIG. 12 were plotted with the proper perspectiveaccording to a particular fan viewing the drawing and aligned andsuperimposed onto an image of the real scene, then it would be trivialto interpret the field locations in all directions which define theplayers defensive range.

Symbols

A common problem shared by most attendees of stadium events is thedifficulty of locating facilities such as the restrooms, concessions, orpolice. Because a large stadium event could be visited by more than100,000 people, it is easy for an individual to become lost.Particularly annoying is the problem of locating one's car after anevent. As the stadium parking lots are generally very large and containmany similar looking cars, it is quite difficult for one to relocatetheir parking place. Since users of the systems of the invention willhave a device capable of locating itself and having previously recordedthe position of the parking space, by pointing and scanning over largeareas of a parking lot, they will be able to look into the display tosee an image of the parking lot with an indication of their parkingspace highlighted. Similarly, the other facilities of the stadium can belocated by looking into the Brighteyes unit. Symbols can be used toindicate locations like a restroom hidden by a wall or image featuresthat may be too small to be seen.

Abstract Symbols

Foul ball planes are abstract in the sense they exist as an invisibleplane but do not represent any physical object. An umpire and otherobservers can judge with some degree of uncertainty when a ball passesthrough a foul ball plane. However, since it is a trivial task to plot aplane onto an image in systems of the invention, the uncertainty of foulball calls can be completely removed. Of course, each user of the visionsystem would see the plane in the correct perspective according to thelocation of the user. It is possible with ball tracking algorithms todetermine the exact moment when the ball passes through the plane and toprovide a signal of that condition. FIGS. 14A and 14B show the path of afoul ball. While watching a ball hit toward the foul ball area of thefield, a tracking system could detect that the ball will likely be afoul ball and indicate an early alarm. The foul ball plane 143, leftinvisible during normal play, could automatically be plotted onto animage of the field in a "warning" color like yellow. As the ball passesthrough the plane FIG. 14B, the color of the plane could change to red144 to indicate a foul ball. Conditions of play could therefore drivethe features being displayed on a Brighteyes unit. Symbols to indicateabstract objects like foul ball planes could be plotted onto real imagesto enhance the information presented to a user.

Ball Tracks

A curveball pitch can be brilliantly executed by some pitchers,sometimes breaking more that twelve inches. Since a baseball is only inone point at one time, it is quite difficult to envision the exact pathof a pitched baseball. If the position of a ball is recorded as the ballapproaches the batter, a plot could be presented to the display whichshows the path of a pitched ball. The maximum lateral displacement couldeasily be computed to determine the breaking quality of the pitch.

Histogram

As each pitch is different, a history of several or many pitches by onepitcher could be displayed simultaneously yielding a histogram ofpitches. Because the interesting property of a fast ball is speedcompared to position, the histogram could be arranged to exclude thosetype of pitches. Displaying the paths of the last three curveballspitched would yield a useful histogram. FIG. 15 shows a pitcher 151, onthe mound 152, the home plate 153 and the batter 154. The last threecurveball 155 pitches can simultaneously displayed as a histogram. Itmay also be instructive to compare the average curveballs of one pitcherverses another. One pitcher may have curve balls that break differentlythat others. These features could be readily illustrated in devices ofthe invention.

Improper Perspective

It may be difficult for some viewers of a curveball pitch to appreciatethe difficulty of hitting the ball. From nearly every view in thestadium, the ball appears to have a trajectory leading into the strikezone. However, the catcher knows differently. He (and the batter) canvery distinctly notice a one foot breaking action.

Since a balls location can be known and mapped with precision during thecourse of a pitch, the path can be plotted onto the coordinate systemfrom any perspective selectable by the user. Of course, a preferredperspective for viewing a curveball pitch is likely the catcher's, anyof an infinite other perspectives can be selected to view the pitch.

If a curveball breaks strongly left-to-right but not at all up-and-down,then the path of the ball could appear to an observer in the stands tobe a straight line (actually parabolic concave down). If one chooses toview the pitch from directly above the field, a position relativelydifficult to realize in an actual game, (although the Goodyear blimp mayargue with this) by selecting that position one can see how much thepitch breaks left-to-right. These are two illustrations where the usersperspective is not preferred, a better perspective is known, and theuser specifies a "virtual position" for displayed images to betranslated.

Panels

Special "windows"-type computer graphical interface objects hereincalled "Panels" are used in images for various purposes includinginformation selection, system command and control. A Panel can consistof a list of related information, selections and/or options. Panels areimage objects that can be related or contain information to some spatialfeature of the game of baseball. A `fixed` Panel is a Panel that isstatic relative to the background or a designated action element. SomePanels may be automatically fixed where the boresight of the unit waspointing when the Panel was accessed, and other Panel may be placed andfixed by the user. If a Panel is to be fixed by the user the Panel wouldappear as a semi-opaque, flashing image on the boresight and wouldremain on the boresight until the user had positioned the Panel at thedesired location and pushed the designate key on the unit. The user mayfix a panel on a specific heading, on a specific heading at a specificrange or at the location of physical object that is moving, if the unitreceives location data for that object from the Action Data System 46,or stationary. The Panel would stop flashing and remain in this fixedposition until moved or closed. To designate a selection in a fixedPanel the user would move the unit as a whole until the desiredselection was located under the crosshair and then push the designatekey on the unit.

Panels may be displayed in the form of a two sided "sign" with "handles"to rotate it about its central horizontal axis. The left "handle" wouldrotate the "sign" 180 degrees to display the other side, the right one90 degrees to present the edge of the "sign" to the user so as totemporarily clear the area for "live" images.

When the Panel was edge-on to the user it would look akin to a barbellwith the "handles" connected by a line and two title bars, one above andone below, floating next to it. The left handle would contain an uparrow, the right a down arrow. By designating either handle and pressingthe designate key the user may rotate the Panel to display either side.To move a Panel, in either case, the user would designate either titlebar and press the designate key thus tying the Panel to the crosshair,move the Panel to the new location and press the designate key againthus fixing the Panel in the new location. To close the Panel, i.e. toremove the Panel from view, the user would place the crosshair over the"Remove" button in the Panel and push the designate key. The unit wouldfix the Remove Panel in front of the user consisting of the questions,"Are you sure?", and the answer options: "Yes", and "No". If "Yes" isselected the Panel in question would be removed from view and the"Remove Panel" would be deleted. If "No" is selected the unit woulddelete the Remove Panel. The unit treats the entire surroundings as a"screen" and the user looks at different parts of the "screen" to locatethe information they desire. Information could be fixed in the sky, onthe back of someone's head, on billboards, any position the user couldsee with their Brighteyes. Panel may be "stacked" over one another. Tomove a Panel to the "top" of a "stack" the user would designate eithertitle bar, as if to move the Panel, and press the designate key again.

Panels may be "attached" and follow the dynamic Action Data Elements.They may be used to display images as well as menu data. FIG. 13 shows a"replay" Panel 131 fixed in the outfield portion of an image of a realscene. The Panel is displaying the play which happened moments ago ontoa "live" image.

Three examples of composite images which are comprised of an image of areal scene and computer generated augmentation are presented as FIGS.17, 18 and 19. FIG. 17 shows defensive players and the playing range ofthose players. FIG. 18 shows an image of computer generated ball tracksrepresenting the last four hits of a particular batter superimposed ontoan image of a baseball field. FIG. 19 shows a computer generated "strikezone" laid onto an image of a real scene. In addition, indicia relatingto the batter may be displayed as text overlaid on an image.

We teach the system operation of a typical embodiment with the help offlow diagrams to guide the reader through relationships of the varioussystems. We present the flow diagrams in various levels, the lownumbered levels corresponding to fundamental processes and the highernumbered levels corresponding to detailed sub-processes. By correctlyplacing all higher level flow diagrams into their appropriate places inlow level diagrams one could form a single flow diagram. The divisiononto many pages is for clarity. There is one continuous flow diagram.The flow diagram shows five modes of operation. The five modes are:initiate, pre-game, game, post game, and shut down. FIG. 16 shows howthe levels cooperate.

INITIATE, FIG. 21

When the Brighteyes unit is first turned on it would go through a seriesof self tests 210 to check its hardware integrity and memory. It wouldthen report unit health to the user and retrieve the hardwareidentification number from internal non-volatile storage. The unit wouldthen request that the user, either orally or by way of a Panel, inputtheir personal identification number 211. The unit would transmit bothID numbers to the Transaction Data System 48 via the two way radio link.

The Transaction Data System 48 would check 212 to see if the unit hadbeen reported as stolen 221. If the unit had been reported stolen theunit would notify security that a suspected stolen Brighteyes unit hadchecked in to the system. The Transaction Data System 48 would bypassthe Personal ID number and credit checks and appear to go into Pre-GameMode 214 as normal. The Transaction Data System 48 would tag the unitnumber and when the unit reported a seat location the Transaction DataSystem 48 would notify security 222 as to the location of the suspectedstolen unit.

If the unit was not reported as stolen the Transaction Data System 48would then recall 223 the user ID number for the unit. If the ID numberthat was inputted by the user did not match 224 the stored number theTransaction Data System 48 would request a new input 225 from the HandHeld Unit twice. After the third attempt the unit would go into Pre-GameMode, bypassing the credit check, but would report to security 226 thatan unauthorized user had checked into the system. The Transaction DataSystem 48 would tag the unit number and when the unit reported a seatlocation the Transaction Data System 48 would notify security as to thelocation of the suspected user.

The Transaction Data System 48 would then check 227 the users credit toensure there were sufficient funds to pay for the air time of the game.If sufficient funds were not available the Transaction Data System 48would ask the unit to request 228 another credit card number from theuser. If none were available the unit would be told to shutdown.

If the hardware ID number and the personal ID numbers matched and therewas sufficient credit available the Transaction Data System 48 wouldcheck 229 to see if it had any record of a ticket for that user for thatspecific game. If there was a ticket number the Transaction Data System48 would transmit the X, Y, Z coordinates of the seat to the Brighteyeshand held unit where it would be stored in the units internal memory andused to generate the units own personal set of coordinates in once theuser was seated.

Once this sub-routine was completed the Transaction Data System 48 woulddownload 213 all the necessary software to run the Baseball Applicationto the unit and the unit would activate Pre-Game Mode 214.

PRE-GAME MODE, FIG. 23

Pre-Game Mode is the mode of operation the unit operates in until theuser changes to the primary mode of operation, Game Mode, and isactivated automatically when the unit is turned on by the user. Pre-GameMode has six Main Menu Options that allow the user to access informationand/or interact with the unit.

Save Arrival Point 230

The Save Arrival Point Main Menu Option allows the user to input thepoint to which they would like to return to after the game is over. Ifthe point is the location of a car the user would input the code for theparking space, i.e. Red, Row 34, Bay 23 by either designating theparking space code or, if the unit cannot process the image, manuallyinputting the code via a Panel. If the point was a bus stop, trainstation, or taxi stand the user would input the Brighteyes location codeof that point by designating the location placard posted at the locationor manually inputting the location code via a Panel if the unit cannotprocess the image. If the unit was equipped with a GPS receiver and thepoint was outdoors the location could be saved as a latitude-longitudeposition. The unit would store the location in its internal memory.

Input Location 231

The Input Location Main Menu Option allows the user to input theirlocation within the stadium before they get to their seat. Locationsaround the outside and concourse of the stadium would have Brighteyeslocation placards, consisting of a number code, posted. The user wouldactivate the Main Menu Option and designate 250 the placard. The unitwould process 251 the image, read the code, transmit the code 252 to theTransaction Data System 48 and would receive 253, from the same system,an X ,Y, Z location in relation to the stadium datum. If the unit cannotprocess the image, the user would input 254 the location code manuallyvia a Panel. If the unit was equipped with a GPS receiver and could seethe sky the location update could be continuous up to the point that theunit loses GPS positioning. If the unit was equipped with an audioin/out system the user could activate the Main Menu Option and read thelocation placard data out loud enabling speech processing software knownto the state of the art input the location code. This may be in additionto or in substitution for other means for defining location within thepreferred embodiment. The unit then stores the acquired location data255. The program flow then returns to the pre game mode main menu 256.

Input Seat Number 232

The Input Seat Number Main Menu Option allows the user to input a newseat number, if the user is a season ticket holder and not sitting intheir usual seat, or input their seat number for that game. The userwould activate the Main Menu Option, hold the ticket up and designate260 the ticket. The unit would process the image 261, read the seatnumber, transmit the number to the Transaction Data System 48 andreceive 262 an X, Y, Z coordinate for that seat number in relation tothe stadium datum. If the unit could not process the ticket image, theuser would manually input 263 the seat number via a Panel. If the unitwas equipped with an audio in/out system the user could activate theMain Menu Option and read the seat number data out loud enabling speechprocessing software to input the location code. This may be in additionto or in substitution for other means for defining location within thebest versions of the system.

Directions 233

The Directions Main Menu Option allows the user to get directions totheir seat from the last saved location. When the Main Menu Option isactivated the unit would check to see if a seat number has been saved270. If one has not then the unit would prompt 271 the user to input theseat number. The unit would then ask 272 if the location has changed. Ifit has, the unit would prompt the user to input 273 the new location.The unit would then ask the user to fix the Directions Panel on aparticular bearing so that as the user moves the Panel would move withthem. The Panel would contain written directions 274 to the inputtedseat number.

Stadium Information 234

The Stadium Information Main Menu Option allows the user to have iconsrepresenting various services available at the stadium fixed over theirposition. When the Main Menu Option was activated the unit would, if notgetting location data from GPS, ask if the location has changed. If ithas, the unit would prompt the user to input the new location asdescribed. The unit would then fix icons representing the closest men'sand ladies toilets, ticket office, security office, etc. at theirlocations in relation to the users position.

GAME MODE, FIG. 29

Game Mode is the primary mode of operation for the baseball application.It is active from the time the user activates it till the time that theuser designates the Post-Game button on the Post-Game panel that theuser fixed after the last out of the game. There are eight Main MenuOptions in Game Mode, each allowing the user to interact with the unitand recall a myriad of information should they so desire. The Game Modeconsists of the following routines: setup 290, auto, coach, scout 291,offense 292, defense 293, pitching 294, teams 295, stadium 296, and FUN!297.

Setup 290

The Setup Main Menu Option allows the user to select and customize theway that basic information about the game is presented. Once seated, theuser would go through the "setup" procedure for that specific game. Theunit would display and fix 310 the Setup Panel in front of the user.This Panel would define how basic information relating to the game wouldbe presented to the user. The Panel would contain a list of options,some of them, if turned on, generating user fixable Panels of their own.For example, a user sitting along the third base line designates the fixscoreline button on the Setup panel. The scoreline would appear in thecenter of the view. The user would then move the entire unit until thescoreline was positioned on the right field wall, press the designatekey and then be able to look away, leaving the scoreline fixed on thewall. If "screen edge" option was designated, the information wouldappear along the bottom of the view. In this way the user can totallycustomize how they view the basic information pertaining to the game.The user could place the scoreline on the fence, the lineups on thebackstop, the names of players left on the bench over the dugout, thenames of players in the bullpen over the bullpen, the batters averageand name over his head, the count, score, outs, and innings over homeplate. This allows totally customized viewing without the need for ascoreboard. All the information, averages, lineups, etc., would bedownloaded by the Transaction Data System 48 to the unit throughout thegame via the digital radio link. This information would be updated bythe Action Data System 46 and would change as the game progressed. Theuser may return to the Setup Main Menu Option at anytime to change theparameters they have previously selected. The user may save theirfavorite setups for each stadium they visit. If the user has a setupsaved the unit would ask if the user would like a previously storedSetup. If the answer is yes then the setup automatically takes place. Ifthe answer is no then the user would go through the setup procedure. Itis possible that there could be several setup options saved for eachstadium from which the user could select. The user may also enter a newseat number in the Setup Main Menu Option if they moved during the game,or the unit was being used by a user in another seat. The user may alsosave the existing set-up at any time by designating save Setup on theSetup Panel. To enter the new seat number the user would select theSetup Main Menu Option, designate the new seat number button and enterthe new seat number. It is also possible that there be a "Multiple User"option in the set up Main Menu Option. If this option were selected bythe user the unit would ask the user "how many users today?". If theuser entered four then each person who would be using the unitthroughout the game would be given a number one through four to identifythemselves to the unit. They would each go through all of the steps inthe "Set Up" Main Menu Option and the unit would remember all of theirindividual parameters. To switch between users the user would locate theMultiple Users Panel and designate the desired number button, or of thenumber of the user who wanted the unit next. As soon as the button wasdesignated the parameters change and the graphics, and Panels would beseen from the point of view of the new user. The user may also selectthe Auto Option 380 which would execute a default setup. The Auto optionmay be turned on and off at will during the game by entering the SetupMain Menu Option.

Coach

The Coach option in the Setup Main Menu Option gives the user access tooptions on defensive and offensive alignments and informed predictionson the play of the game. When Coach is activated the user would fix theCoach Panel in the desired location. The Panel would consist of aselection for each team. When a team is designated Coach depending onwhether that team is batting or fielding, would explain the defensivealignment of the fielders or predict the probable offensive strategy ofthe team. When the Coach is on he will also interject during the gamee.g. "Watch the guy on first, he's a flyer". The information could bedisplayed in the left side of the view as a speech bubble coming from acaricature coach, or the report may be in the form of a full motion andsound film clip. The speech bubble would stay on the screen for a finiteperiod and then disappear. The information would be stored either in theunit itself or in the stadium database and would be compiled throughinterviews with and analysis of past games by baseball experts.

Scout 291

The scout option in the Setup Main Menu Option would give the useraccess to expert opinion on the player's or team's offensive ordefensive abilities. When the user designates the Scout a Panel thatdefines which player and what type of report would appear in front ofthe user. The user would then fix the Panel and select the desiredoption by moving the unit as a whole, placing the crosshair over theoption and pressing the designate key. The "Visual Desig." option wouldbe automatically active, until "Other" was designated, with thecrosshair flashing. The user may then either select a player by placingthe crosshair on that player, if he is visible, or designating his namein one of the player lists, lineup, bench, bullpen, that could have beenplaced around the park in the setup stage and then pushing the designatekey. If the user selects "other" a new Panel would appear and fix itselfin front of the user. The Panel would also enable the user to select aplayer from a team roster that has not already been fixed.

If Home or Visiting team is selected the complete roster of that teamwould fix itself in front of the user, the user would designate whichplayer with the crosshair and the Panel would close. If the user selects"other team" the logos of all teams would fix themselves, ordered bydivision and record, in front of the user. The user would then designatethe appropriate team and go through the steps previously described. Bychoosing one of the options the user would receive either an offensiveor defensive scouting report on the desired player or a short biographyof the player. The report would either appear on the left side of thescreen and remain there until the user closed the report or be in theform of a film clip of a coach, for example, with full motion and sound.Depending upon the memory size of the unit, the information for thescouting reports could be stored within the unit or accessed from theTransaction Data System 48, via the two-way radio link, as needed. Avariation of this would be to store the reports on the two teams playingthe game in the unit and only access the Transaction Data System 48 forinformation on other teams. An example of an offensive scouting reportwould be, "Dead pull hitter, can't lay off the high outside fastball."An example of a defensive report would be, "Covers a lot of ground buthas a weak arm." The information would be stored either in the unititself or in the Transaction Data System 48 and would be compiledthrough interviews with and analysis of past games by baseball experts.

Auto Option 380

The Auto Main Menu Option would allow the user to take advantage off allthe different aspects of the Brighteyes Baseball Application withouthaving to touch another key or switch on the unit. They simply have tolook through the unit at the appropriate moment, and at the appropriateplace to see the pre-programmed information, in the form of text orgraphics, the unit would be displaying for the user. The unit wouldeither access information it needed from it's own memory, or download itvia the two way digital radio link. The Auto Option would still allowthe user to set seat location, so it could be changed throughout thegame. All the Main Menu Options with the exceptions of Setup and Autowould be locked out until Auto had been deactivated. In the Setup MainMenu Option the only active options would be those relating to new ormultiple seat numbers. If the unit was equipped with GPS and wasreceiving positional data from this system the Setup Main Menu Optionwould also be locked out until Auto was deactivated. To input a new oradditional seat number the user would select the temporary Seat MainMenu Option. The same method of seat selection described would be used.The "Seat" option would not be necessary if the unit was equipped withGPS, and was operating in a stadium that had GPS pseudo satellites. Theonly option left in the main menu in this case would be "Set Up". Theunit would display a set selection of Panel and icons throughout theentire game. The user would go through the placement process to fix eachPanel in the desired location in the stadium. This allows the user tocustomize viewing for their individual point of view.

All the information would be constantly updated and displayedautomatically by the unit. The unit would receive information from itsmemory, and transmissions from the stadiums Action and Transaction DataSystem 48 48s needed for the display. When the Auto Main Menu Option isdeactivated the unit reverts to the set-up that was active at the timeAuto was activated.

Offense 292

The "Offense" Main Menu Option gives the user access, both graphicallyand statistically, to information regarding Offense play about theselected player or team. When the user selects 400 the "Offense" MainMenu Option by using the scrolling key and pressing the Main Menu Optionactivate key the user would asked to fix 401 the "Offense" Panel wheredesired. On one side this Panel would allow the user to define 402 theplayer or team they are interested in, turn on or off the statistics andgraphics options and the "Statistics Parameters" sub-Panel.

On the other side of the "Offense" Panel the statistics as defined inthe "Statistics Parameters" sub-Panel would be displayed along with aBatting/Running button. Batting statistics would consist of RBIs, runs,batting avg., hits, home runs, et cetera Running statistics wouldconsist of stolen bases, attempts, runs, caught stealing, etc. Asdifferent players where designated the Panel would change to display thestatistics of the designated player.

If "Area Graphics" 411 was turned on the unit would display the area ofthe field that the ball is most likely to be hit into. This area wouldbe viewed from the users perspective and its location would be definedby the parameters set in the "Offensive Graphics Main" Panel.

If "Full Graphics" 412 was turned on the unit would show, in the form ofcolored lines defining the path of the ball from the batters strikezone, all the hits and outs, and where and how far they go, as definedby the parameters in the "Offensive Graphics Main" Panel. The view ofthese lines would be specific to the perspective of the user. The usercould see the tracks of the hits/outs as they would appear from hisspecific position.

When either of the Graphics options was turned on for the first time,the unit would ask the user to fix 413 the "Offense Graphics Main"Panel. The best location for this Panel would probably be "stacked" withthe "Offense" Panel. This Panel consists of all the different parametersthat affect the offensive graphics displayed. The sub-Panel in thisPanel would cover parameters ranging from the time period to pitchlocation and include a default setting. When the user has made thedesired selections, or changes, the user would select OK and the unitwould display the appropriate graphics.

The balltracks or lines would be stored in the unit or in the stadiumdatabase. They would be stored as digital information and would beentered either by an ultra fine radar tracking system in each stadium,or by one or several data entry personnel at the stadium as the gameprogressed. Initially the tracks would have to be entered manually toget the database going. As soon as the database was operational, havingall information, statistics, results, balltracks for players still inthe league, ball tracks for great players from the past, etc. for theentire history of baseball, then the information would be entered up tothe second in each stadium across the country or world. The databases ineach stadium would be linked to each other and they would all receivethe same information.

The information needed for the displayed items, statistics, balltrajectories, etc., would be either stored in the memory of the unit,or, in the case of a unit with limited memory, downloaded via the twoway digital radio link from the stadium database when requested by theuser. As soon as the user had finished using information downloaded viathe digital radio it could delete it from it's memory and if deleted theunit would have to access the database again if the same information wasneeded by the user.

Defense 293

The "Defense" Main Menu Option gives the user access, both statisticallyand graphically, to defensive information about a selected player orteam. When the user selects 541 "Defense" the user would fix 542 the"Defense" Panel. On one side this Panel would allow the user to definewhich player or teams statistics will be displayed, turn on or off thestatistics and graphics options and set 551 the "Statistics Parameters"sub-Panel. On the other side the statistics as defined by the selectionswould be displayed 552.

Defensive statistics could consist of chances, errors, assists, fielding%, throwing arm of player, etc. If the player selected is a catcher thenumber of runners thrown out stealing and % thrown out would be includedin the statistics list. All statistics would be in the unit memory, ordown loaded as needed from the stadium database. As different playerswhere designated the Panel would change to display the statistics of thedesignated player.

If "Graphics" is turned on, the unit would graphically represent, in theform of `domes`, the defensive range of the selected players accordingto the selections made in the "Graphics Parameters" Panel. The defensiverange of a player is defined by the players physical ability along withdistance from the plate and velocity of the batted ball. This isillustrated in FIG. 10. Before the pitch is thrown, the defensive bubblewould be displayed as defined in the "Graphics Parameters" Panel. If thevelocity of the struck ball can be measured, and the data sent to theunit, in near real time the defensive bubble would shrink or growaccordingly. When "Graphics" turned on for the first time the unit wouldask the user to fix "Graphics Parameters" Panel. This Panel would allowthe user to select the players who have their bubbles displayed and alsoselect the strength of the hit. This Panel, would define theconfiguration of the defensive bubbles before the ball was struck. Theinformation would be stored either in the unit itself or in the stadiumdatabase and would be compiled through interviews with and analysis ofpast games by baseball experts.

Pitching 294

The "Pitching" Main Menu Option gives the user access, both graphicallyand statistically, to pitching information about the selected player orteam. When the user selects the "Pitching" Main Menu Option the userwould be asked to fix the Pitching Panel. On one side this Panel wouldallow the user to define the pitcher or team they are interested in,turn on or off the graphics and statistics options and set the"Statistics Parameters" sub-Panel.

On the other side of the Pitching Panel the statistics as defined in theStatistics Parameters sub-Panel would be displayed. Pitching statisticswould consist of ERA, won-lost record, strike outs, walks, wild pitches,innings pitched, run against, saves, appearances, etc. As differentpitchers where designated the Panel would change to display thestatistics of the designated pitcher.

If "Area Graphics" was turned on the unit would display the area of thefield that the ball is most likely to be hit into. This area would beviewed from the users perspective and its location would be defined bythe parameters set in the "Pitching Graphics Main" Panel.

If "Full Graphics" was turned on the unit would show, in the form ofcolored lines defining the path of the ball from the batters strikezone, all the hits and outs against the selected pitcher, and where andhow far they go, as defined by the parameters in the "Pitching GraphicsMain" Panel. The view of these lines would be specific to theperspective of the user. i.e. The user could see the tracks of thehits/outs as they would appear from his specific position.

When either of the "Graphics" is turned on for the first time the unitwould ask the user to fix the "Pitching Graphics Main" Panel. This Panelwould probably be "stacked" with the "Pitching" Panel. This Panelconsists of all the different parameters that affect the pitchinggraphics displayed. The sub-Panel in this Panel would cover parametersranging from time period to pitch location and playing conditions andinclude a default setting.

When the user has made the desired selections, or changes, the userwould select OK and the unit would display the appropriate graphics.

The information needed for the displayed items, statistics, balltracks,et cetera, would be either stored in the memory of the unit, or, in thecase of a unit with limited memory, downloaded via the two way digitalradio link from the stadium database when requested by the user. As soonas the user had finished using information downloaded via the digitalradio it would delete it from it's memory. The unit would have to accessthe database again if the same information was needed by the user.

Teams 295

The "Teams" Main Menu Option gives the user access to, and the abilityto fix around the stadium, lists of players or team schedules. Theselists may also be used to designate players for other Main Menu Optionssuch as Offense and Pitching. If the user selects "Teams" the unit woulddisplay and the user would fix a Panel, the "Team Select" Panel.

If the user selects "Other Team" the unit would fix the logos of allteams, ordered according to division and standing, as a Panel in frontof the user. On one side of the Panel the National League would bedisplayed and on the other side the American League. The user would thenselect a team, the Logos Panel would be deleted and the user fix thatteams entire roster as a Panel. If the home or visiting team wasselected the user would fix the Roster Type Panel consisting of"Lineup", "Bullpen", "Bench" and "Schedule" buttons. The user wouldselect the desired list and the selected list would be fixed as a Panelby the user. If "Schedule" was selected the teams schedule, includinginformation on previous wins and losses, times of games etc. wouldappear as a Panel and be fixed by the user.

Stadium 296

The Stadium Main Menu Option gives the user access to information aboutstadium services, amenities and infrastructure and visually displays theinformation to the user. If the user selects "Stadium" the unit woulddisplay the Stadium Main Panel which would be fixed where desired. ThisPanel would contain all the different categories available to the user.

As the various category buttons on the Stadium Main Panel are designatedicons representing the closest locations in the selected categories,based upon the selections made in Stadium Options, would be displayed ordeleted throughout the stadium. These icons would be displayed eitherdirectly over the location, if it is visible, or over the nearest doorto the actual location of the selection with appropriatearrows/directions. If the user selected the "Stadium Options" Main MenuOption the unit would display, as a Panel to be fixed by the user, the"Stadium Options" Panel listing all the option in each category. Bymaking the appropriate selections the user can define what is displayedin the "Stadium" Main Menu Option.

FUN! 297

When the Fun! Main Menu Option is activated the user would be asked tofix the Fun! Main Menu Panel. On one side the panel would contain theinteractive sub-panels and on the other the reenactment sub-panel.

Interactive Betting and Gambling

The "Interactive" sub-panels allow the user to try and predict what isgoing to happen during the game.

The Real Money sub-Panel gives the user the choice of betting realmoney, in states where this is legal of course, or "Fun Money" by way ofa switch set to one or the other. The unit would know if this type ofgambling was legal in that state and if it was not would not display theReal Money sub-Panel, instead automatically setting "Fun Money" as thecoinage for the interactive sub-Panel. If "Fun Money" is selected theuser is given a set amount of "fun dollars" which may be wagered in thegames. At the end of the game the three Brighteyes user at the game withthe most "fun dollars" would win prizes. If real cold hard cash wasselected the unit would check the users credit, via the Transaction DataSystem 48 and inform them of their limit. If there was insufficientcredit available the unit would inform the user and switch to Fun Money.All losses and gains would be credited or debited to or from theappropriate Real or Fun Money accounts and would be displayed on thescreen edge.

The Bet sub-Panel allows the user to set amount they would bet perselection. To set an amount the user would `click` on the differentnotes until the desired total is reached, i.e. to bet fifteen dollarsthe `five` would be designated three times. The amount would remainunaltered until the user selected "Change" and then reentered a newamount. The on/off switch in the real money option is to enable thegambler to take a break, or practice for a while, and play the gameswithout any wagering. The reason for not having this switch in Fun Moneyoption is that the user has nothing real to lose and everything to gain.

The "Ghost Fielder" sub-Panel gives the user access to the first game.To play the user would switch on, set a bet amount per fielder moved,designate "Select" and then designate which fielder or fielders theyfeel are in the wrong position. The user would then position `ghostimages` of these fielders in the positions on the field they desire anddesignate the "OK" button. The unit, or central database, would thentrack the next ball in fair ground. If the ball passed through theghosts `defensive bubble` and not through the real players bubble theuser would win. If the ball passed through the ghosts `defensive bubble`and through the real players bubble the user would lose. If the ballpassed through the real players bubble and not through the ghosts theuser would lose. If the ball passed through the real fielders bubble andalso through another unselected fielders bubble the bet would be off. Ifthe ball passed through neither bubble the bet would be off. After eachfair ball the ghost images would disappear. This game would requiretracking of the ball, including altitude.

The "Pick-a-Patch" sub-Panel would give the player access to a game thatlets them try to predict where the next fair ball would first hit theground or if caught where the fielder is either standing or lands. Toplay the player would turn on the sub-Panel, set a bet amount ifdifferent from that displayed, designate the Pick-a-Patch button andthen pick their "patch" by targeting the desired area with the crosshairand pressing the designate key. Only one patch may picked per play andit would have to be at least partially in fair territory. Once theplayer is satisfied with the patch they have picked they would designatethe OK button in the associated Panel. The "patch" would be representedby a 5' diameter, at field level, red circle centered on the selectedposition as it would be seen relative to the user. The odds wouldincrease the further from home plate the patch was. If the ball orplayer who caught the ball was in their patch the player would win.

The "Predict Score" sub-Panel allows the player to try and predict thefinal outcome of the game, the winning pitcher and the player thatdrives in the winning run. The player may bet on all or some of theoptions with the same or different amounts as they see fit. To select afinal score the user would designate the score button. A Panelcontaining the team names and numbers one to nine would be fixed infront of the user, the user would designate each team name and score anddesignate "OK". The selected score would be entered as the prediction.The score prediction could be changed for the first three innings, withlower and lower odds, and at the end of the third would be fixed. Thenames of the winning pitcher and player who drove in the winning runwould be entered by designating the appropriate button and designatingthe name from rosters fixed in front of the user in the normal way. Eachplayers odds would be different according to their seasons performancein the appropriate categories. The names could only be put in before thefirst pitch. If the final score was predicted, before the game, and thecorrect two names were also picked, the Trifecta, the user would win asubstantial prize.

The Fun Main Menu Option would also contain a reenactment sub-panel.This sub-panel would contain buttons for historical events that havetaken place in that stadium and buttons for different groupings ofreenactments. When the History button is designated the unit woulddisplay or delete icons around the stadium that mark a historic spot inthe stadiums history. For example, there is a history icon located inthe left field bullpen area. The user would place the crosshair of theirunit over the icon and press the designate key. The icon would change toa description of the historical significance of the spot, a "Re-enact"button and a `Done` button. In this example the description would read,"Hank Aaron's 715th home run to break the Babe Ruth's record. Hit onApr. 8th, 1974". If the user designated the "Re-enact" button the userwould see, from their perspective, a computer animated version of theactual event. The animated sequence would be down loaded from theTransaction Data System 48 and the unit would block out all presentplayers and replace them with images generated by the computer. When there-enactment was started by designating home plate the animationsequence would run, complete with audio. They would see Hank Aaron hitthat home run, and it would be seen by them as if they were there,occupying the very seat that they were sitting in for the present game.Not the same old footage run on television. The user could view theaction again by designating the "Re-enact" button, or close thedescription by designating the "Done" button. The description woulddisappear and the icon would reappear over the historic spot. The numberand placement of these icons would depend on which stadium you were at.Chicago's Wrigley field would have a great deal of historic icons due tothe age if the stadium, but a fairly new stadium like Baltimore's CamdenYards would have relatively few historical icons.

The re-enact sub-panel would also have several buttons, such as the"This Day in Baseball" button, that when designated would ask the userto fix a panel containing related reenactment buttons. If this buttonwas designated the unit, knowing what date it was by referring to it'sinternal clock, or querying the Transaction Data System 48, would listseveral historic events that had taken place that date. The user woulddesignate an event button and a "Description" Panel for that event wouldappear on the screen. The "Description" Panel would be exactly like thehistorical "Description" Panel with a "Re-enact" button and a "Close"button. If the user chose to re-enact the event the database would download the computer animated sequence complete with audio. Thesurroundings would gradually change, as viewed through the unit, and theuser would find themselves seated in either a totally different stadium,but in a seat with the exact view of the field as their view of the realgame, or in the same stadium years ago in the same seat. The playingfield, especially the infield diamond, would still be the real surfacethey were seeing that day, and the animation would take place on top ofthat surface. If the user were seated in a seat that did not exist inthe stadium they appear to be in they would see the re-enactment as ifthey were floating in mid air. If their seat position was located suchthat their ability to see the playing field was blocked, then theseobstructions would automatically be removed from the field of view sothe user's perspective of the playing field would remain the same. Theuser would watch the event until the sequence was complete and then theview would morph back to the present day view of the stadium. The usercould either view the sequence again, or close the description with theclose button located on the description Panel. There would also be otherbuttons that covered other types of plays such as Homers of the month,Pitching magic and Flying fielders that would function exactly the sameway as "This day in Baseball".

Once an option had been designated the unit would fix a Panel listingthe individual plays on the bore sight in front of the user and the userwould designate the desired play. After the desired play had beendesignated the Panel would be deleted and the unit would fix the"Description" Panel for that play, and re-enact the designated play fromthe users perspective if desired.

The Fun Main Menu Panel would also have a "Fantasy Player" sub-panel.This would allow the user to make an existing player appear to look likea great player from the past, uniform and all. The user could, forinstance, make it look like Willie Mays, the "Say Hey Kid", was playingcenter, instead of some green rookie who had just been called up fromthe minor leagues. It could not make the player play any differently, itwould just make him appear, only while viewed through the Brighteyesunit, to be someone else. The user would designate the "Fantasy Player"option and a Panel containing a list of players from the past would befixed by user. The list would include a button for each name. The userwould designate a player from the list. The crosshairs would now beblinking, and the user would designate the real player they wished tocover up with their Brighteyes. The player designated would be replacedby the player previously chosen from the list. The user could accessinformation about the Fantasy Player just as they could a normal player.To cancel the effect the user would return to the Fantasy Player Paneland switch the player's name off.

POST GAME MODE, FIG. 86

Post Game Mode is active from the time that the Post Game Button on thePost Game Panel in Game Mode has been designated, and confirmed, untilthe time that unit is turned off. There are seven Main Menu Options inPost Game Mode. The Post Game Mode consists of the following routines:Transport, Box Score, Standings, Next Game Info., Stadium, Game Replays,and Location.

Transport

The Transport Main Menu Option gives the user access to directions tothe various methods of transport available at that stadium. When theMain Menu Option is activated, the unit will ask the user to fix theTransport Panel on bearing only. This is so that the Panel will stay inthe same place in relation to the user when the user moves. The user mayask for directions to the stored arrival point (see Pre Game Mode), thenearest taxi stand or, if available, bus stops or trains. To receivedirections the user simply designates the desired selection. If arrivalpoint or taxi was designated the appropriate directions wouldappropriate directions would appear on the other side of the TransportPanel, once again on bearing only. This Panel would contain a list ofavailable buses or trains. When a specific bus or train has beendesignated the directions to that site would be displayed of the "backside" of the Transport Panel as before. All directions would be from thelast saved location.

Box Score

The Box Score Main Menu Option allows the user to fix the box score ofthe game as a Panel to be looked at later.

Standings

The Standings Main Menu Option allows the user to fix the current leaguestandings as a Panel, National on one side and American on the other.

Next Game

The Next Game Main Menu Option Allows the user to fix next gameinformation as a Panel. This information would be: location, start time,opposing team, and anticipated pitchers.

Stadium

The Post-Game Mode Stadium Main Menu Option functions in exactly thesame way as the Game Mode Stadium main Menu Option.

Replays

The Replays Main Menu Option gives the user access to a list of replaysfrom the just completed game. When the option is activated the userwould fix the Replay Panel. This Panel would contain the list of replaysavailable. By designating the desired replay the user would see the playrun again from the perspective of their seat during the game or, in thecase of multiple users, from the perspective of the last user. Note thatthe user does not have to be in their seat to see a replay but they mustbe within range of the Transaction Data System 48 radio link.

Location

The Location Main Menu Option allows the user to input their locationaround the stadium. The Post Game Mode Location Main Menu Optionfunctions exactly as the Pre-Game Mode Location Main Menu Option does.

Although the present invention has been described in considerable detailwith reference to certain preferred versions thereof, other versions arepossible. Therefore, the spirit and scope of the appended claims shouldnot be limited to the description of the preferred versions presentedhere. In particular, it would be an obvious extension of the inventionto other sports which are enjoyed by spectators.

What is claimed is:
 1. An electro-optic vision system comprising:acamera for producing an electronic image of a scene being addressed; aposition and attitude determining means for determining the position andattitude of the vision system; a data store responsive to outputs ofposition and attitude determining means, the data store havinginformation stored therein; a computer operable for combining the storedinformation with the electronic image to produce an augmented image; andan electro-optic display operable for displaying the augmented image. 2.A vision system of claim 1, further comprising transducers incommunication with the computer, the transducers being fixed about anevent forum and directed to make measurements from objects within afield of interest whereby said measurements are converted intoinformation which relates to the scene being addressed.
 3. An apparatusof claim 2, said transducers being operable for measuring the dynamicposition of an element from the group; players; baseball; coaches;referees; umpires; bat, in play.
 4. A vision system of claim 1, saidposition determining means being further comprised of a plurality oftransducers at fixed locations about the event forum operable fortransmitting a signal which enables said computer to determine theposition of the camera.
 5. A vision system of claim 1, furthercomprising an electro-optic display operable for displaying theaugmented image, said position determining means being further comprisedof a second data store having prior knowledge of seating locations aboutan event stadium, and a means to enter a spectator seat number.
 6. Avision system of claim 1, said data store further comprising recordeddata which represents events from a previous time in a format that canbe combined with images of a real scene being addressed by said camera.7. A vision system of claim 1, said data store further comprisingrecorded data which represents symbols in a format that can be combinedwith the images of a real scene being addressed by the camera.
 8. Avision system of claim 1, said electro-optic display further beingoperable for displaying the augmented image, said data store furthercomprising recorded data which represents statistical information in aformat that can be combined with the images of a real scene beingaddressed by the camera.
 9. A vision system of claim 1, said data storefurther comprising recorded data which represents action traces in aformat that can be combined with the images of a real scene beingaddressed by the camera.
 10. A vision system of claim 1, said data storefurther comprising recorded data which represents computer graphicalinterface devices in a format that can be combined with the images of areal scene being addressed by the camera.